Skip to content

fix(cache): keep the per-turn billing line out of the cached prefix (+3 attribution fixes) - #161

Open
qbq-leo-martens wants to merge 7 commits into
teamchong:mainfrom
qbq-leo-martens:fix/cache-attribution
Open

fix(cache): keep the per-turn billing line out of the cached prefix (+3 attribution fixes)#161
qbq-leo-martens wants to merge 7 commits into
teamchong:mainfrom
qbq-leo-martens:fix/cache-attribution

Conversation

@qbq-leo-martens

Copy link
Copy Markdown

Why

Running Claude Code through pxpipe, my prompt-cache hit rate sat at ~0% for a whole
working day while the dashboard cheerfully reported ">95%". That turned out to be two
separate things: one real cache buster, and three places where the telemetry could not
have shown it.

All four are small, independent commits.

1. fix(cache): the per-turn billing line lived inside the cached prefix

Claude Code (>= 2.1.x) folds an x-anthropic-billing-header: cc_version=... line into
its system text, as an unmarked trailing block. That value is not session-stable —
it carries a per-turn id — so the tail of the cached prefix changed on every single
request
: guaranteed full cache miss, every turn, every session.

stripBillingLine() already existed, but it only ever ran on the concatenated
rawSysText, and by then the block Claude Code actually sends had already been
classified as part of the prefix, so the regex never matched what mattered.

Fix: liftBillingLineFromSystem() pulls the line out of every system block before
the static/dynamic split, and the proxy re-attaches it as a real request header
(x-anthropic-billing-header) — where it belongs, and where it cannot invalidate
anything. A block that held nothing but the billing line disappears instead of leaving an
empty shell behind.

Escape hatch: PXPIPE_KEEP_BILLING_LINE=1 restores the old in-prompt placement.

Effect on my host: cache reads went from ~0 to the expected 90%+ of input tokens per
session, immediately and across restarts.

2. fix(cache): bust attribution digested the boundary message, not the MARKED span

cachePrefixMarkedSha / cachePrefixMarkedBytes were computed over the boundary message
rather than the span that actually carries the cache_control marker — i.e. the bytes
Anthropic really caches. The diagnostics therefore could never point at what changed
inside the pinned prefix (which is exactly what I needed to find #1). Now the digest
covers the marked span and the marker position is recorded alongside it, with per-layer
digests (tools / system / head) so a bust can be attributed to one layer.

3. fix(cache): history-image hash was read from messages[0]

history_image_sha8 was taken from messages[0] instead of the message that actually
carries the imaged history, so a perfectly stable history looked like it churned every
turn (and real churn could hide).

4. fix(dashboard): the by-events cache-hit rate divided by a field that doesn't exist

renderStatsTableFragment() divides cacheHitEvents by s.eventsWithBaseline, but
stats.ts emits that number as eventsWithUsage. undefined > 0 is false, so the
event-based hit rate silently rendered - forever. Renamed the field in
FullStatsSummary to match the producer.

Tests

  • new tests/cache-bust-attribution.test.ts (10 cases), including the regression that
    matters: two turns differing ONLY in the billing id keep a byte-identical cached
    prefix
    , plus per-layer digest movement and marked-span stability while the live tail
    grows.
  • extended tests/dashboard-api.test.ts and tests/render.test.ts.
  • full suite: 931 passing, tsc --noEmit clean, npm run build clean.

Notes

Found while debugging a real ~2h token burn on a self-hosted box; happy to adjust naming,
split the commits differently, or drop #4 into its own PR if you prefer.

qbq-leo-martens and others added 7 commits July 30, 2026 21:52
…essages[0]

`historyImageSha8` hashed the image blocks on `messages[0]`, described in its
own docstring as "the synthetic history message". It is not: whenever a slab
anchor exists, `collapseHistory` returns `[...head, syntheticUser, ...tail]`
and transform.ts passes `protectedPrefix = slabAnchorIdx + 1`, so on every
Claude Code request `messages[0]` is the protected slab message. The field
that exists to prove history-image byte-identity was therefore reporting SLAB
stability under the history name — a drifting collapse boundary still showed a
rock-steady `history_image_sha8` in events.jsonl, which is precisely the
signal teamchong#11 attribution relies on. Locate the synthetic message by its banner,
the same way `cachePrefixDigest` does.

Also split the pinned-prefix digest per layer. `cache_prefix_sha8` proves THAT
the cacheable prefix moved but never WHICH layer moved, and the layers fail for
different reasons and need different fixes: tools drift when the client loads a
deferred tool, system drifts when volatile text (env, git status) rides inside
the pinned span, and the imaged head drifts when a collapse boundary or marker
placement moves. Emit `cache_prefix_tools_sha8` / `cache_prefix_system_sha8` /
`cache_prefix_head_sha8` alongside the aggregate so one session of telemetry
names the culprit instead of narrowing it to "somewhere in the prefix".

Motivation: a production session showed 530/530 requests with a unique
`cache_prefix_sha8` at a constant `cache_prefix_bytes`, 0 cache_read across
87M cache_create tokens — with the aggregate hash alone the cause could not be
attributed, and `history_image_sha8` looked stable because it was measuring
the wrong message.
…t attribution

`cachePrefixDigest` hashed every block up to and including the message that
carries pxpipe's imaged prefix. That is not the span Anthropic caches: caching
ends at the LAST cache_control marker, and after a history collapse the two
differ by construction. The synthetic message's newest freeze chunk re-renders
on every turn BY DESIGN (append-only rendering: only completed chunks are
frozen) and it sits AFTER the pinned marker — so the boundary-scoped digest
changes every single turn even when the cached span is byte-stable, and
`cache_prefix_sha8` reads "pxpipe busted its own cache" on healthy traffic.

Emit `cache_prefix_marked_sha8` / `_marked_bytes` / `_marker_pos` alongside:
the digest through the breakpoint, its size, and where the breakpoint sits as
`m<msg>.b<block>`. The marked digest is the one that must hold turn over turn;
the marker position makes a roaming breakpoint — a bust cause in its own right,
previously invisible — a one-field diagnosis.

Tests pin the contract that decides whether cache_read happens at all: two
consecutive turns of one session keep the marked span byte-identical while the
live tail grows.
… text

Claude Code folds `x-anthropic-billing-header: <...>` into the *first* system
block, and that line carries a per-turn request id. Anything inside the
cache_control-anchored prefix has to be byte-identical across turns, so this
one line rewrote the pinned prefix on every single request and cost us the
whole prompt cache — every turn paid cache_create instead of cache_read.

transform now strips the line out of the system text (leaving no empty block
behind) and hands it back on TransformInfo; proxy sets it as the outbound
header it always claimed to be. Set PXPIPE_KEEP_BILLING_LINE=1 to fall back
to the old in-prompt placement.

Also records the two digests that made this diagnosable: a text-only system
digest and per-block `index[*]:len:hash4` fingerprints, which is what pinned
the churn to block 0 rather than to the collapse boundary.
…xists

renderStatsTableFragment read s.eventsWithBaseline, but stats.ts emits
eventsWithUsage; the missing field made the comparison false for every
session, so the "cache hit (by events)" row silently rendered "-" no matter
what the cache did. The type carried the stale name too, so tsc had no
chance to catch it.

Test pins the value to its own row (the table renders as one line, so a
bare toContain passes on a neighbouring number) and fails with '-' against
the old expression.
The 100-image limit is a property of the wire, not of pxpipe: it counts
the images the client sent (pasted screenshots, pictures a tool returned)
together with every image we add. We priced only our own, so a request
that arrived already full got imaged further and came back 400/500 —
two sessions became unresumable.

Every imaging path now spends from one shared headroom:

  imageHeadroom() = cap - margin - ours - theirs

and the caller's images are counted once, before any rewrite, at both
nesting levels (top-level and inside tool_result).

The gate is quantitative, not boolean. A first cut asked "is there ANY
room left?" and then emitted a whole 15-page slab into it: 94 client
images + a 400k slab put 109 images on the wire. Now:

  - the slab must fit whole or not at all — imaging half of it would
    re-key the cache prefix on every turn whose client-image count moved;
  - each tool_result pages against the LIVE headroom, not a fixed 10;
  - after rendering we verify the real page count, because paging is
    budgeted at denseGeo.cols while rendering happens at o.cols;
  - the history collapse is skipped entirely at zero headroom — its
    budget floors at 1, so it would otherwise emit exactly one image
    and still fail the request.

Degrading means "do not emit this turn", never "un-emit": each turn is
re-derived from the client's own text transcript, so our images are
never stranded there.

Telemetry: native_images and image_budget_skips reach the event log, and
passthrough_reasons loses an allow-list that swallowed exactly the two
reasons worth diagnosing (kept_sharp, image_budget).

Measured, 300 turns + 60k slab + 60k tool_result:

  clients= 0 | slab=3 toolres=3 hist=50 | wire= 56 | text  233k
  clients=50 | slab=3 toolres=3 hist=44 | wire=100 | text  206k
  clients=80 | slab=3 toolres=3 hist=11 | wire= 97 | text  809k
  clients=90 | slab=3 toolres=1 hist= 0 | wire= 94 | text 1053k

Client images always win; we shrink to fit. What remains is a cliff, not
a slope: the collapse is our only reducer and it is image-based, so at a
full wire we drop from compressed straight to raw.
…ge count

Three gaps, all of the same kind: the code knew something and never said it.

markCacheDead() existed but nothing ever called it. A rejected request never
populated a prefix cache, so the append-only freeze it was protecting protects
nothing — but only the transform side knew that, and the proxy never told it.
responseLeftNoCache() now names the rule and the response path applies it:
413, any 5xx, and a 400 whose body says the prompt is too long. Everything
else stays warm on purpose. A 401, a 404, a rate limit say nothing about the
cache, and guessing "cold" there would re-cut a live grid and burn the whole
prefix as cache_create — the exact cost this module exists to avoid.

imageCount is what we RENDERED, and the wire disagrees: the history collapse
replaces whole messages, so a tool_result image inside the collapsed range is
never sent. Measured on a tool-heavy shape: 91 rendered, 49 on the wire.
info.wireImages now counts the outgoing body itself, and the event carries it
whenever it differs from the render count.

The event log gains history_freeze_step, history_budget_trimmed and
history_pack_fill, so the adaptive grid is finally observable: until now
nothing recorded whether it had coarsened, why, or whether a repack landed on
a cache we believed dead.

The dashboard says the same in words, but only when there is something to say:
"6 images came from your side and count against the same 100-image request cap
· 48 rendered pages never went out — the history collapse absorbed those
messages (49 on the wire)". A quiet turn stays quiet.

Not fixed here: those 48 absorbed pages are content loss, not just miscounting.
An imaged tool_result serializes into the history as "[tool_result]\n[image]",
so the tool output survives only as the factsheet's exact identifiers. The
cure is ordering — collapse before imaging — which changes the outgoing bytes
for every session and re-keys every cache once, so it wants its own change.
The history serializer renders an image block as the literal string `[image]`.
Tool_results were imaged first and collapsed second, so any tool output old
enough to fall into the collapsed prefix reached the model as:

    [tool_result]
    [image]

The output itself was gone — only the fact sheet's exact identifiers survived.
The slab was already shielded from precisely this ("collapsing it would reduce
slab images to [image] placeholders and destroy the cache anchor"); the same
reasoning was never extended to tool_results.

Swapping the two steps means the collapse serializes the caller's original
text, and only what stays live gets imaged. Measured across three shapes
(200/100/400 turns, a tool_result every 3rd-6th):

    chars carried into the history images   21k  ->  2062k
    pages rendered but never sent            6   ->     0
    outgoing text                          2226k ->  1712k   (-23%)

Less outgoing text is not a rounding artifact: with the tool output imaged
away, the collapse saw a nearly empty transcript and judged collapsing
unprofitable — on the 400-turn shape it did not run at all. Feeding it the
real text lets it do its job.

The block moved verbatim; only its indentation and the two order comments
changed. tests/collapse-order.test.ts pins the property rather than the
ordering (does tool output reach the model at all), and both of its core cases
fail against the previous order.

The wire count now equals the render count, so the disagreement recorded in
tests/native-image-cap.test.ts is gone; that test flips to asserting no page is
paid for and then dropped.

Bytes on the wire change for every session with history, so the first request
after this lands as cache_create once.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants